Migration API for record-based entities #1241

dmitrykuzmin · 2020-02-28T18:48:02Z

This PR extends the RecordBasedRepository with API to perform the migration of stored entities to account for the model changes.

Migration API

The repository is extended with applyMigration(ID, Migration) and applyMigration(Set<ID>, Migration) methods.

The Migration is a transformation operation which accepts entities one-by-one and modifies their state, lifecycle flags or can even delete the entity record. This class (or rather its descendants, ProjectionMigration and ProcessManagerMigration) can be inherited to create custom user-defined migration operations.

The applied entity modification can be configured by implementing the method apply(entityState):

class ProjectMigration extends ProjectionMigration<...> {

    // A mapper from the "old" to a "new" state.
    public Project apply(Project state) {
        return state.toBuilder()
                    .setName("new_name")
                    .build();
    }
}

The new state returned from apply will then be injected into the entity under migration.

In apply(...), it's also possible to call additional modification methods like markArchived() or removeFromStorage(). See Migration for details.

Each migration is preceded by the entity lookup by ID and is finalized by either saving the modified entity back to the repository or deleting the entity record, if it is configured by the Migration.

All changes are applied to entity under an active Transaction, with the last step of migration being a transaction commit. This means all entity lifecycle events are triggered normally, enabling subscription updates and other system-events-dependent features.

The posted events will have an instance of newly introduced MigrationApplied system event as a producing message.

Columns update

The standard migrations provided by Spine include the UpdateProjectionColumns and UpdatePmColumns migrations for projections and process managers respectively. When applied, such migration triggers the recalculation of interface-based columns of an entity, propagating the new values to the storage fields as well as to the serialized entity state.

The column update can be applied as follows:

// Projection.
projectionRepository.applyMigration(ImmutableSet.of(id1, id2), new UpdateProjectionColumns<>());

// Process manager.
pmRepository.applyMigration(ImmutableSet.of(id1, id2), new UpdatePmColumns<>());

Other migrations

The framework also provides Mark(Projection/Pm)Archived, Mark(Projection/Pm)Deleted and Remove(Projection/Pm)FromStorage migrations which can mark entity archived, mark entity deleted and physically remove an entity record from the storage respectively. The migrations are available for both projections and process managers under the respective migration packages.

Other

This PR also fixes an issue with Nullable interface-based columns propagation and improves the description of Columns.valuesIn(...) operation.

Version

Spine version advances to 1.4.11.

…into fix-interface-based-columns � Conflicts: � license-report.md

An issue described by the TODO is rather serious and requires further discussion and probably changes of several public and internal API elements. So it's better off being converted into a GitHub issue.

dmitrykuzmin · 2020-02-28T19:05:16Z

@armiol, @yuri-sergiichuk PTAL.

server/src/main/java/io/spine/server/entity/Migration.java

server/src/main/java/io/spine/server/entity/RecordBasedRepository.java

yuri-sergiichuk · 2020-03-01T23:41:29Z

server/src/test/java/io/spine/server/projection/given/Migrations.java

+import io.spine.server.projection.Projection;
+import io.spine.server.projection.ProjectionMigration;
+
+public final class Migrations {


I'd say these migrations could be part of the Framework standard migrations. Worth considering moving them to the sources?

Also document them.

Also documented the method better.

The `MarkArchived`, `MarkDeleted` and `RemoveFromStorage` migrations are now exposed as standard Spine migrations for both projections and process managers. `ProjectionColumnsUpdate` and `ProcessManagerColumnsUpdate` renamed to `UpdateColumns`.

dmitrykuzmin · 2020-03-02T14:10:57Z

@yuri-sergiichuk PTAL again.

server/src/main/java/io/spine/server/entity/EntityLifecycle.java

yuri-sergiichuk · 2020-03-02T14:25:34Z

server/src/main/java/io/spine/server/entity/EntityLifecycle.java

+     * @return the event or an empty {@code Optional} if the posting was blocked by the
+     *         {@link #eventFilter}
+     */
+    public final Optional<Event> onMigrationApplied() {


That's a bit strange that we're able to filter out the migration event, but on the other hand, in the Migration we throw an IllegalStateException in case of such filtering.

armiol

@dmitrykuzmin LGTM in general. Please see my suggestions though.

armiol · 2020-03-02T14:58:02Z

server/src/main/java/io/spine/server/entity/Migration.java

+ * {@link MigrationApplied} event as the producing message.
+ *
+ * <p>To create a user-defined {@code Migration} in real life scenarios, consider inheriting from
+ * {@link io.spine.server.projection.ProjectionMigration} and


Please use an alt text to shorten the links. E.g. {@link io.spine.server.projection.ProjectionMigration ProjectionMigration}.

server/src/main/java/io/spine/server/entity/Migration.java

armiol · 2020-03-02T15:02:02Z

server/src/main/java/io/spine/server/procman/migration/UpdateColumns.java

+ * A migration operation that does the update of interface-based columns of a
+ * {@link ProcessManager}.
+ *
+ * <p>When applied to an entity, this operation will trigger the recalculation of entity storage


Please append a paragraph explaining when such migration may be handy. I.e. describe a use-case scenario.

armiol · 2020-03-02T15:02:32Z

server/src/main/java/io/spine/server/projection/migration/UpdateColumns.java

+/**
+ * A migration operation that does the update of interface-based columns of a {@link Projection}.
+ *
+ * <p>When applied to an entity, this operation will trigger the recalculation of entity storage


Same as above.

armiol · 2020-03-02T15:05:11Z

server/src/main/java/io/spine/server/projection/migration/UpdateColumns.java

+ * @see io.spine.server.entity.RecordBasedRepository#applyMigration(Object, Migration)
+ */
+@Experimental
+public final class UpdateColumns<I,


Can we have a single UpdateColumns class, which would be created using a factory method?

E.g. UpdateColumns.of(MyProjectView.class) or UpdateColumns.of(MyProcess.class).

We then would detect if it is a projection or a process manager and create a corresponding instance.

It would be cool to avoid two public classes with the same name.

Now in such case the framework will just log a warning.

The old standard migrations are now renamed to preserve naming uniqueness (e.g. `procman.MarkArchived` -> `MarkPmArchived`) but are also made `Internal`.

The approach with delegating migrations doesn't seem to work unless we expose even more `Migration` methods as `protected` `Internal` which is undesirable.

yuri-sergiichuk

@dmitrykuzmin LGTM

The `EntityDeleted` event message is extended with `deletion` one-of which allows to determine whether the entity was just marked as deleted or physically removed from the storage.

…into extend-migration-api � Conflicts: � license-report.md � pom.xml � version.gradle

dmitrykuzmin · 2020-03-04T14:07:34Z

@armiol PTAL again.

yuri-sergiichuk and others added 30 commits February 20, 2020 13:09

Make sure interface-based columns work with entity and not its state

b672bf5

Bump version

0becc30

Add ability to apply basic migration operations to repository entities

c4ec78e

Make the Migration doc less restrictive

76506ea

Add tests for column update operation

78828a5

Revert the interface-based column values recalculation

768ce7e

Merge branch 'master' of https://github.com/SpineEventEngine/core-java …

2bfecb8

…into fix-interface-based-columns � Conflicts: � license-report.md

Handle null values returned from interface-based column in a safer way

4de26bf

Polish the docs

bb191a6

Improve formatting

beab63b

Expand the Migration doc

2d5da6f

Fix grammar

83bb5ef

Fix formatting

7946bea

Fix grammar

f004528

Improve doc

09c3444

Improve doc of column update operations

2e62cb3

Add missing checkNotNull assertions

4889e0c

Fix grammar

ef909d7

Pull applyMigration(id, migration) method down to record based repo

b2c63a5

Mark public API as Experimental

018141e

Add new Migration API and provide basic implementation for columns

0798f61

Remove unused imports

ca80b35

Return a posted event from the system write side

ea026e0

Fix erroneous doc

8ead2ad

Fix a compilation error

f95a91e

Remove a TODO

a9b36b6

An issue described by the TODO is rather serious and requires further discussion and probably changes of several public and internal API elements. So it's better off being converted into a GitHub issue.

Do not complete migration sooner even if entity is marked for removal

fbf33f3

Do not complete migration sooner even if entity is marked for removal

ced3f38

Reorganize the Migration code to shorten the long code blocks

5f75aa9

Add artificially injected last handled message to acknowledged signals

b4fecd7

dmitrykuzmin requested review from armiol and yuri-sergiichuk February 28, 2020 19:04

yuri-sergiichuk reviewed Mar 1, 2020

View reviewed changes

dmitrykuzmin added 4 commits March 2, 2020 14:38

Mark package-private methods that may look like public API Internal

f912e7c

Also document them.

Fix how Migration.finishCurrentOperation() is applied

71affa9

Also documented the method better.

Extend standard Spine migrations

92c5d4a

The `MarkArchived`, `MarkDeleted` and `RemoveFromStorage` migrations are now exposed as standard Spine migrations for both projections and process managers. `ProjectionColumnsUpdate` and `ProcessManagerColumnsUpdate` renamed to `UpdateColumns`.

Polish the docs

08b192e

yuri-sergiichuk reviewed Mar 2, 2020

View reviewed changes

Document the modified EntityLifecycle methods

df9cd49

armiol reviewed Mar 2, 2020

View reviewed changes

dmitrykuzmin added 6 commits March 2, 2020 18:46

Remove exception in case the event filter blocks MigrationApplied

07f72ce

Now in such case the framework will just log a warning.

Shorten doc links

dd3be7c

Mark the method as Internal

628104b

Improve doc of UpdateColumns

47e6a40

Expose standard migrations as single delegating migration instances

34f99ba

The old standard migrations are now renamed to preserve naming uniqueness (e.g. `procman.MarkArchived` -> `MarkPmArchived`) but are also made `Internal`.

Remove delegating migration and just rename the standard migrations

b548475

The approach with delegating migrations doesn't seem to work unless we expose even more `Migration` methods as `protected` `Internal` which is undesirable.

yuri-sergiichuk reviewed Mar 3, 2020

View reviewed changes

dmitrykuzmin added 4 commits March 4, 2020 15:38

Post an EntityDeleted event if record is removed from the storage

1635af3

The `EntityDeleted` event message is extended with `deletion` one-of which allows to determine whether the entity was just marked as deleted or physically removed from the storage.

Merge branch 'master' of https://github.com/SpineEventEngine/core-java …

a78c6a1

…into extend-migration-api � Conflicts: � license-report.md � pom.xml � version.gradle

Update license report

bb6c7ef

Fix a typo

e66f4f7

dmitrykuzmin requested a review from armiol March 4, 2020 14:07

armiol approved these changes Mar 4, 2020

View reviewed changes

dmitrykuzmin merged commit 93040b9 into master Mar 4, 2020

dmitrykuzmin deleted the extend-migration-api branch March 4, 2020 14:21

This was referenced Mar 8, 2020

Release 1.5.0 #1247

Merged

Add callback to Repository to allow erasure of entities #906

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration API for record-based entities #1241

Migration API for record-based entities #1241

dmitrykuzmin commented Feb 28, 2020 •

edited

Loading

dmitrykuzmin commented Feb 28, 2020

yuri-sergiichuk Mar 1, 2020

dmitrykuzmin commented Mar 2, 2020

yuri-sergiichuk Mar 2, 2020

armiol left a comment

armiol Mar 2, 2020

armiol Mar 2, 2020

armiol Mar 2, 2020

armiol Mar 2, 2020

yuri-sergiichuk left a comment

dmitrykuzmin commented Mar 4, 2020

Migration API for record-based entities #1241

Migration API for record-based entities #1241

Conversation

dmitrykuzmin commented Feb 28, 2020 • edited Loading

Migration API

Columns update

Other migrations

Other

Version

dmitrykuzmin commented Feb 28, 2020

yuri-sergiichuk Mar 1, 2020

Choose a reason for hiding this comment

dmitrykuzmin commented Mar 2, 2020

yuri-sergiichuk Mar 2, 2020

Choose a reason for hiding this comment

armiol left a comment

Choose a reason for hiding this comment

armiol Mar 2, 2020

Choose a reason for hiding this comment

armiol Mar 2, 2020

Choose a reason for hiding this comment

armiol Mar 2, 2020

Choose a reason for hiding this comment

armiol Mar 2, 2020

Choose a reason for hiding this comment

yuri-sergiichuk left a comment

Choose a reason for hiding this comment

dmitrykuzmin commented Mar 4, 2020

dmitrykuzmin commented Feb 28, 2020 •

edited

Loading